A stapler, hole punch and staple remover are necessary items for office work. Such items individually would occupy substantial desk space. Therefore, a multi-purpose item that can perform all of these functions is desirable to eliminate such shortcomings. A U.S. Pat. No. 4,002,281 was granted to the present inventor for a stapler having a shiftable stapling blade with a staple remover and a hole punch mounted on a same. This structure is by no means perfect, because the shiftable blade therein is liable to glide back and forth after a considerable time of use when the slide button of the shifter becomes loose. Thus, there is still a need for an improved multi-purpose stapler.